Updating maps based on traffic object detection

ABSTRACT

Systems, methods, and computer-readable media are provided for receiving traffic object data from a plurality of autonomous vehicles, the traffic object data including a geographic location of a traffic object, comparing the traffic object data of each of the plurality of autonomous vehicles with known traffic object data, determining a discrepancy between the traffic object data of each of the plurality of autonomous vehicles and the known traffic object data, grouping the traffic object data of each of the plurality of autonomous vehicles based on the determining of the discrepancy between the traffic object data of each of the plurality of autonomous vehicles and the known traffic object data, determining whether a group of traffic object data of the grouping of the traffic object data of each of the plurality of autonomous vehicles exceeds a threshold; and updating a traffic object map based on the traffic object data of the group that exceeds the threshold.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit of U.S.application Ser. No. 16/774,794, filed on Jan. 28, 2020, entitledUPDATING MAPS BASED ON TRAFFIC OBJECT DETECTION, which is expresslyincorporated by reference herein in its entirety.

BACKGROUND 1. Technical Field

The subject technology provides solutions for updating maps based onobject detection and in particular, for updating traffic maps forautonomous vehicles based on discrepancies of a traffic object detectedwith a camera.

2. Introduction

An autonomous vehicle is a motorized vehicle that can navigate without ahuman driver. An exemplary autonomous vehicle can include varioussensors, such as a camera sensor, a light detection and ranging (LIDAR)sensor, and a radio detection and ranging (RADAR) sensor, amongstothers. The sensors collect data and measurements that the autonomousvehicle can use for operations such as navigation. The sensors canprovide the data and measurements to an internal computing system of theautonomous vehicle, which can use the data and measurements to control amechanical system of the autonomous vehicle, such as a vehiclepropulsion system, a braking system, or a steering system. Typically,the sensors are mounted at fixed locations on the autonomous vehicles.

In an autonomous vehicle environment, the environment itselfcontinuously changes over time, which may dictate updates to internalmaps of the autonomous vehicle environment. For example, traffic objectposition (e.g., traffic light, traffic signal, or traffic sign), type,and lane association can change abruptly due to construction or a caraccident. If traffic maps are not updated accordingly with informationreceived from sensors of the autonomous vehicles in a fleet, there isthe risk of an autonomous vehicle entering an intersection when theautonomous vehicle does not have the “right of way.” For example,according to the autonomous vehicle's current map, a stop sign should beat an intersection, when in actuality, the stop sign has been replacedwith a traffic signal because of construction. Without an updatedtraffic map, the autonomous vehicle would enter the intersectionthinking that it had the right of way, when in actuality, the autonomousvehicle is running a red traffic signal.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain features of the subject technology are set forth in the appendedclaims. However, the accompanying drawings, which are included toprovide further understanding, illustrate disclosed aspects and togetherwith the description serve to explain the principles of the subjecttechnology. In the drawings:

FIG. 1 illustrates an example autonomous vehicle environment including acomputing system in communication with an autonomous vehicle;

FIG. 2 is a block diagram of an example sensor positioning platform formechanically moving, rotating, and/or positioning a payload of sensorson an autonomous vehicle, in accordance with some examples;

FIG. 3 illustrates an example method for localizing a vehicle, inaccordance with some examples;

FIGS. 4A and 4B illustrate example methods for managing traffic signallocation discrepancies, in accordance with some examples; and

FIG. 5 is a schematic diagram of an example computing devicearchitecture, in accordance with some examples.

DETAILED DESCRIPTION

Various examples of the present technology are discussed in detailbelow. While specific implementations are discussed, it should beunderstood that this is done for illustration purposes only. A personskilled in the relevant art will recognize that other components andconfigurations may be used without parting from the spirit and scope ofthe present technology. In some instances, well-known structures anddevices are shown in block diagram form in order to facilitatedescribing one or more aspects. Further, it is to be understood thatfunctionality that is described as being carried out by certain systemcomponents may be performed by more or fewer components than shown.

The disclosed technologies address a need in the art for improvements invehicle sensor technologies and capabilities. In some examples, a sensorpositioning platform on an autonomous vehicle can include multipleco-located sensors that can be dynamically rotated or repositioned foroptimal sensor coverage. The sensors can be mounted on a rotating sensorcarrier structure of the sensor positioning platform, which functions asan azimuth positioning stage for the sensors. The sensor positioningplatform can include a motor for moving, repositioning, and/or rotatingthe sensors and sensor carrier structure, and electrical components forcontrolling the movement, repositioning, and/or rotation of the sensorsand sensor carrier structure through the motor. The sensor positioningplatform can also include a sensor cleaning system that allows thesensors in the rotating sensor carrier structure to be cleaned asneeded, a cable management system that interconnects the sensors withother electrical components of the autonomous vehicle and provides afreedom of movement that enables the sensors to remain connected whenrotating, and other components as described herein.

The sensor positioning platform can receive (e.g., from a computingsystem on the autonomous vehicle) commands for moving, repositioning,and/or rotating the sensors and sensor carrier structure. Thus, throughthe sensor positioning platform, the sensors can be repositioned toincrease sensor coverage, provide instantaneous field of view, andtarget specific areas or objects. The sensors can also be repositionedto account for changes in the vehicle's motion, driving angles anddirection, as well as relative changes in the vehicle's environment andthe motion, angle, and position of surrounding objects. The dynamic andadaptable sensor repositioning herein can improve the sensors'visibility, accuracy, and detection capabilities. The sensorrepositioning platform can allow autonomous vehicles to monitor theirsurroundings and obtain a robust understanding of their environment. Thesensor repositioning platform and associated functionality can alsoprovide benefits in cost, sensor data redundancy, and sensor fusion.

FIG. 1 illustrates an example autonomous vehicle environment 100. Theexample autonomous vehicle environment 100 includes an autonomousvehicle 102, a remote computing system 150, and a ridesharingapplication 170. The autonomous vehicle 102, remote computing system150, and ridesharing application 170 can communicate with each otherover one or more networks, such as a public network (e.g., a publiccloud, the Internet, etc.), a private network (e.g., a local areanetwork, a private cloud, a virtual private network, etc.), and/or ahybrid network (e.g., a multi-cloud or hybrid cloud network, etc.).

The autonomous vehicle 102 can navigate about roadways without a humandriver based on sensor signals generated by sensors 104-108 on theautonomous vehicle 102. The sensors 104-108 on the autonomous vehicle102 can include one or more types of sensors and can be arranged aboutthe autonomous vehicle 102. For example, the sensors 104-108 caninclude, without limitation, one or more inertial measuring units(IMUs), one or more image sensors (e.g., visible light image sensors,infrared image sensors, video camera sensors, surround view camerasensors, etc.), one or more light emitting sensors, one or more globalpositioning system (GPS) devices, one or more radars, one or more lightdetection and ranging sensors (LIDARs), one or more sonars, one or moreaccelerometers, one or more gyroscopes, one or more magnetometers, oneor more altimeters, one or more tilt sensors, one or more motiondetection sensors, one or more light sensors, one or more audio sensors,etc. In some implementations, sensor 104 can be a radar, sensor 106 canbe a first image sensor (e.g., a visible light camera), and sensor 108can be a second image sensor (e.g., a thermal camera). Otherimplementations can include any other number and type of sensors.

The autonomous vehicle 102 can include several mechanical systems thatare used to effectuate motion of the autonomous vehicle 102. Forinstance, the mechanical systems can include, but are not limited to, avehicle propulsion system 130, a braking system 132, and a steeringsystem 134. The vehicle propulsion system 130 can include an electricmotor, an internal combustion engine, or both. The braking system 132can include an engine brake, brake pads, actuators, and/or any othersuitable componentry configured to assist in decelerating the autonomousvehicle 102. The steering system 134 includes suitable componentryconfigured to control the direction of movement of the autonomousvehicle 102 during navigation.

The autonomous vehicle 102 can include a safety system 136. The safetysystem 136 can include lights and signal indicators, a parking brake,airbags, etc. The autonomous vehicle 102 can also include a cabin system138, which can include cabin temperature control systems, in-cabinentertainment systems, etc.

The autonomous vehicle 102 can include an internal computing system 110in communication with the sensors 104-108 and the systems 130, 132, 134,136, and 138. The internal computing system 110 includes one or moreprocessors and at least one memory for storing instructions executableby the one or more processors. The computer-executable instructions canmake up one or more services for controlling the autonomous vehicle 102,communicating with remote computing system 150, receiving inputs frompassengers or human co-pilots, logging metrics regarding data collectedby sensors 104-108 and human co-pilots, etc.

The internal computing system 110 can include a control service 112configured to control operation of the vehicle propulsion system 130,the braking system 132, the steering system 134, the safety system 136,and the cabin system 138. The control service 112 can receive sensorsignals from the sensors 104-108 can communicate with other services ofthe internal computing system 110 to effectuate operation of theautonomous vehicle 102. In some examples, control service 112 may carryout operations in concert with one or more other systems of autonomousvehicle 102.

The internal computing system 110 can also include a constraint service114 to facilitate safe propulsion of the autonomous vehicle 102. Theconstraint service 116 includes instructions for activating a constraintbased on a rule-based restriction upon operation of the autonomousvehicle 102. For example, the constraint may be a restriction onnavigation that is activated in accordance with protocols configured toavoid occupying the same space as other objects, abide by traffic laws,circumvent avoidance areas, etc. In some examples, the constraintservice 114 can be part of the control service 112.

The internal computing system 110 can also include a communicationservice 116. The communication service 116 can include software and/orhardware elements for transmitting and receiving signals to and from theremote computing system 150. The communication service 116 can beconfigured to transmit information wirelessly over a network, forexample, through an antenna array or interface that provides cellular(long-term evolution (LTE), 3rd Generation (3G), 5^(th) Generation (5G),etc.) communication.

In some examples, one or more services of the internal computing system110 are configured to send and receive communications to remotecomputing system 150 for reporting data for training and evaluatingmachine learning algorithms, requesting assistance from remote computingsystem 150 or a human operator via remote computing system 150, softwareservice updates, ridesharing pickup and drop off instructions, etc.

The internal computing system 110 can also include a latency service118. The latency service 118 can utilize timestamps on communications toand from the remote computing system 150 to determine if a communicationhas been received from the remote computing system 150 in time to beuseful. For example, when a service of the internal computing system 110requests feedback from remote computing system 150 on a time-sensitiveprocess, the latency service 118 can determine if a response was timelyreceived from remote computing system 150, as information can quicklybecome too stale to be actionable. When the latency service 118determines that a response has not been received within a thresholdperiod of time, the latency service 118 can enable other systems ofautonomous vehicle 102 or a passenger to make decisions or provideneeded feedback.

The internal computing system 110 can also include a user interfaceservice 120 that can communicate with cabin system 138 to provideinformation or receive information to a human co-pilot or passenger. Insome examples, a human co-pilot or passenger can be asked or requestedto evaluate and override a constraint from constraint service 114. Inother examples, the human co-pilot or passenger may wish to provide aninstruction to the autonomous vehicle 102 regarding destinations,requested routes, or other requested operations.

As described above, the remote computing system 150 can be configured tosend and receive signals to and from the autonomous vehicle 102. Thesignals can include, for example and without limitation, data reportedfor training and evaluating services such as machine learning services,data for requesting assistance from remote computing system 150 or ahuman operator, software service updates, rideshare pickup and drop offinstructions, etc.

The remote computing system 150 can include an analysis service 152configured to receive data from autonomous vehicle 102 and analyze thedata to train or evaluate machine learning algorithms for operating theautonomous vehicle 102. The analysis service 152 can also performanalysis pertaining to data associated with one or more errors orconstraints reported by autonomous vehicle 102.

The remote computing system 150 can also include a user interfaceservice 154 configured to present metrics, video, images, soundsreported from the autonomous vehicle 102 to an operator of remotecomputing system 150, maps, routes, navigation data, notifications, userdata, vehicle data, software data, and/or any other content. Userinterface service 154 can receive, from an operator, input instructionsfor the autonomous vehicle 102.

The remote computing system 150 can also include an instruction service156 for sending instructions regarding the operation of the autonomousvehicle 102. For example, in response to an output of the analysisservice 152 or user interface service 154, instructions service 156 canprepare instructions to one or more services of the autonomous vehicle102 or a co-pilot or passenger of the autonomous vehicle 102.

The remote computing system 150 can also include a rideshare service 158configured to interact with ridesharing applications 170 operating oncomputing devices, such as tablet computers, laptop computers,smartphones, head-mounted displays (HMDs), gaming systems, servers,smart devices, smart wearables, and/or any other computing devices. Insome cases, such computing devices can be passenger computing devices.The rideshare service 158 can receive from passenger ridesharing app 170requests, such as user requests to be picked up or dropped off, and candispatch autonomous vehicle 102 for a requested trip.

The rideshare service 158 can also act as an intermediary between theridesharing app 170 and the autonomous vehicle 102. For example,rideshare service 158 can receive from a passenger instructions for theautonomous vehicle 102, such as instructions to go around an obstacle,change routes, honk the horn, etc. The rideshare service 158 can providesuch instructions to the autonomous vehicle 102 as requested.

The remote computing system 150 can also include a package service 162configured to interact with the ridesharing application 170 and/or adelivery service 172 of the ridesharing application 170. A useroperating ridesharing application 170 can interact with the deliveryservice 172 to specify information regarding a package to be deliveredusing the autonomous vehicle 102. The specified information can include,for example and without limitation, package dimensions, a packageweight, a destination address, delivery instructions (e.g., a deliverytime, a delivery note, a delivery constraint, etc.), and so forth.

The package service 162 can interact with the delivery service 172 toprovide a package identifier to the user for package labeling andtracking. Package delivery service 172 can also inform a user of whereto bring their labeled package for drop off. In some examples, a usercan request the autonomous vehicle 102 come to a specific location, suchas the user's location, to pick up the package. While delivery service172 has been shown as part of the ridesharing application 170, it willbe appreciated by those of ordinary skill in the art that deliveryservice 172 can be its own separate application.

One beneficial aspect of utilizing autonomous vehicle 102 for bothridesharing and package delivery is increased utilization of theautonomous vehicle 102. Instruction service 156 can continuously keepthe autonomous vehicle 102 engaged in a productive itinerary betweenrideshare trips by filling what otherwise would have been idle time withproductive package delivery trips.

FIG. 2 is a block diagram of an example sensor positioning platform 200for mechanically moving, rotating, and/or positioning sensors 104-108 ona sensor carrier structure 220 implemented by the autonomous vehicle102. The sensor positioning platform 200 can be attached to, coupledwith, and/or otherwise secured to the autonomous vehicle 102. The sensorcarrier structure 220 with the sensors 104-108 can be situated outsideof the autonomous vehicle 102 in order to have access to, and/orvisibility into, the external or outside environment (e.g., outside orexternal to the autonomous vehicle 102) so the sensors 104-108 cancapture sensor data or measurements pertaining to the outsideenvironment, conditions or characteristics of the outside environment,objects or humans located in the outside environment, etc.

In addition to providing the sensors 104-108 access to, and/orvisibility into, the external or outside environment, as furtherdescribed herein, the sensor positioning platform 200 can mechanicallymove, rotate, and/or reposition the sensor carrier structure 220 toallow the sensors 104-108 on the sensor carrier structure 220 to capturesensor data or measurements for different areas or regions of theoutside environment, extend the addressable field of regard, extendand/or provide an instantaneous field of view, provide sensor visibilityor access into a focused or specific area or object, account fordifferent angles, account for different vehicle maneuvers, etc. Thesensor data or measurements can be used to detect objects (e.g., othervehicles, obstacles, traffic signals, signs, etc.), humans, animals,conditions (e.g., weather conditions, visibility conditions, trafficconditions, road conditions, etc.), route or navigation conditions,and/or any other data or characteristics associated with the outsideenvironment.

In some examples, the autonomous vehicle 102 can use the sensor data ormeasurements to perform (or when performing) one or more operations,such as mapping operations, tracking operations, navigation or steeringoperations, safety operations, braking operations, maneuvers, etc. Forexample, the autonomous vehicle 102 can use the sensor data ormeasurements to gain insight or visibility into the outside environmentand the outside environment conditions. The autonomous vehicle 102 canthen use such insight when making navigation decisions, such asdetermining a velocity, determining a maneuver, determining how to avoidan object, determining a trajectory, determining navigation changes(e.g., changes in position, velocity, angle, direction, etc.), and soforth.

The sensor positioning platform 200 can include a base 202. The base 202can include an actuator system 204 and one or more rotary cableassemblies 216. Moreover, the one or more rotary cable assemblies 216can include power and/or communication lines (e.g., cables, wires,flexible printed circuits, etc.) for supplying power to the sensors104-108 on the sensor carrier structure 220 and carrying data to and/orfrom the sensors 104-108.

In some examples, the one or more rotary cable assemblies 216 can feedpower and communication lines to the sensors 104-108 for powering thesensors 104-108 and communicatively connecting (directly or indirectly)the sensors 104-108 to the internal computing system 110 and/or theactuator system 204, while allowing the sensors 104-108 to have freedomof movement in order to rotate with the sensor carrier structure 220while receiving power and remaining communicatively connected to theinternal computing system 110 and/or the actuator system 204.

In some cases, the one or more rotary cable assemblies 216 can includedata lines that connect the sensors 104-108 to a communications device244 and/or an image processing engine 246. The data lines can allow thesensors 104-108 to communicate with the communications device 244 tosend and receive data signals (e.g., sensor data, instructions,commands, information, etc.) to and from the communications device 244.Moreover, the data lines can allow image sensors (106, 108) on thesensor carrier structure 220 to provide, to the image processing engine,image data (e.g., images, videos, frames, etc.) captured by such imagesensors.

The communications device 244 can include, for example and withoutlimitation, a network interface, a switch, a hub, a relay/proxy, or anyother network device capable of switching, forwarding, and/or routingdata. In some implementations, the communications device 244 can supportnetwork communications over or across one or more networks, such as aprivate network (e.g., a LAN), a public network (e.g., a WAN, a cloudnetwork, etc.), a hybrid network, etc. For example, the communicationsdevice 244 can support wireless communications, such as cellularcommunications, WIFI communications, etc.; wired or cablecommunications, such as Ethernet communications, fiber opticcommunications, etc.; and/or any other type of communications.

The communications device 244 can be communicatively connected with theinternal computing system 110 and/or any other computing device, and canthus send and/or receive data to and/or from the internal computingsystem 110 and/or any other computing devices. Thus, the communicationsdevice 244 can route or forward data between the sensors 104-108 and theinternal computing system 110 (or any other computing device). Moreover,in some cases, the communications device 244 can be part of, orimplemented by, the internal computing system 110.

The image processing engine 246 can be part of, or implemented by, theinternal computing system 110 or a separate computing device. Moreover,in some cases, the image processing engine 246 can be part of, orimplemented by, a same computing system as the communications device244. For example, both the image processing engine 246 and thecommunications device 244 can be part of, or implemented by, theinternal computing system 110 or a separate computing device.

The image processing engine 246 can receive image data (e.g., images,frames, videos, etc.) from image sensors (e.g., 106, 108) on the sensorcarrier structure 220 and perform one or more image processing and/orpre-processing operations on the image data, such as, for example andwithout limitation, filtering, scaling, sub-sampling, color correction,color conversion, geometric transformations, noise reduction,demosaicing, spatial filtering, image restoration, image enhancement,frame rate conversion (e.g., up-conversion, down-conversion),segmentation, feature extraction, etc. The image processing engine 246can then provide the processed image data to the internal computingsystem 110 for further use, processing, analysis, etc.

The actuator system 204 can be configured to control a position, angle,and/or movement of the sensor carrier structure 220 and the sensors104-108 on the sensor carrier structure 220. For example, the actuatorsystem 204 can include a motor 212 for controlling the positioning,rotation, and/or movement of the sensor carrier structure 220 hostingthe sensors 104-108, as further described herein. The motor 212 on theactuator system 204 can receive, from a motor controller 240, a commandinstructing the motor 212 to move or rotate the sensor carrier structure220 with the sensors 104-108 to a specific angle and/or position inorder to change the angle, position, and/or field-of-view (FOV) of thesensors 104-108 on the sensor carrier structure 220.

In some examples, the motor 212 can be an electrical motor capable ofconverting electrical energy into mechanical energy that the motor 212can use to move the sensor carrier structure 220 and the sensors 104-108in the sensor carrier structure 220. In some implementations, the motor212 can be a gimbal motor. Moreover, the motor controller 240 caninclude one or more electronic circuits (e.g., one or moremicroprocessors, microcontrollers, central processing units (CPUs),graphics processing units (GPUs), digital signal processors (DSPs),and/or any other suitable electronic circuits or hardware), and/or caninclude and/or can be implemented using computer software, firmware, orany combination thereof, to perform the various operations describedherein.

In some examples, the motor controller 240 can include one or morecomputing and/or electronic components, such as one or more CPUs,Input/Output (I/O) ports or peripherals, timers, memories (e.g.,electrically erasable programmable read-only memory (EEPROM), read-onlymemory (ROM), random-access memory, and the like), controllers,processors, storage devices, and/or any other electronic circuits orhardware. Moreover, the motor controller 240 can include memory (notshown), such as EEPROM, for storing data, firmware, software, and/or anycombination thereof.

The motor controller 240 can send control signals to the motor 212 tomove, rotate, and/or control the motor 212, which can then move, rotate,and/or position the sensor carrier structure 220 with the sensors104-108 to a specific position, angle, and/or location. In some cases,the motor controller 240 can generate the control signals based on,and/or in response to, one or more commands or instructions received bythe motor controller 240 from the internal computing system 110 on theautonomous vehicle 102. For example, the internal computing system 110can send commands or instructions to the motor controller 240 formechanically moving, rotating, and/or positioning the sensor carrierstructure 220 with the sensors 104-108 and/or the motor 212 on thesensor positioning platform 200. The motor controller 240 can receivesuch commands or instructions, parse the commands or instructions,generate one or more control signals based on the commands orinstructions, and send the one or more control signals to the motor 212on the actuator system 204, which can cause the motor 212 to move thesensor carrier structure 220 with the sensors 104-108 to a specificposition, angle, and/or location.

In some cases, when generating control signals, the motor controller 240can calculate a difference between a requested position (e.g., specifiedin the commands or instructions received from the internal computingsystem 110) of the motor 212 (and the sensor carrier structure 220 withthe sensors 104-108) and an actual or current position of the motor 212(and the sensor carrier structure 220 with the sensors 104-108). Forexample, the motor controller 240 can obtain sensor data from a positionsensor 208 in the actuator system 204, which can include measurements ofa current or actual position of the motor 212, and use such measurementsto determine a current or actual position of the motor 212. The motorcontroller 240 can use the current or actual position of the motor 212to calculate an error or difference between the current or actualposition of the motor 212 and the requested position for repositioningthe motor 212 (and the sensor carrier structure 220 with the sensors104-108).

The motor controller 240 can then use the calculated error or differenceto make any adjustments to the position defined in the one or morecontrol signals for the motor 212. In some cases, the motor controller240 can continuously receive position measurements from the positionsensor 208 to calculate such errors or differences, and make adjustmentsto the position specified in the control signals to the motor 212. Thisway, the motor controller 240 can fine tune the position specified inthe control signals to the motor 212 to account for any such errors andincrease an accuracy of the position adjustments of the motor 212 (andthe sensor carrier structure 220 of sensors 104-108).

The position sensor 208 used to obtain position measurements for themotor 212 can include one or more sensor devices, which can include anytype of sensor, encoder, transducer, detector, transmitter, and/orsensing component capable of measuring the position (e.g., linear,angular, etc.) and/or change of position of a target or object, such asthe motor 212. Non-limiting examples of position sensors (208) that canbe used to obtain position measurements (e.g., displacement, linearposition, angular position, etc.) for the motor 212 include opticalencoders, potentiometers, magnetic position sensors (e.g., Hall effectsensors, magnetorestrictive position sensors, etc.), rotary encoders,linear encoders, capacitive position sensors, inductive position sensors(e.g., resolvers, linearly variable differential transformers, etc.),fiber-optic position sensors, photodiode arrays, incoders, etc. Theseexamples are not exhaustive and are simply provided for explanationpurposes, as any other types of position sensors are also contemplatedherein.

The position sensor 208 can reside under the motor 212, along an outsideof the motor 212, along an outside of a rotor of the motor 212, along anoutside of a stator of the motor 212, and/or in any other location thatallows the position sensor 208 to obtain positioning measurements forthe motor 212 and fit within an assembly (e.g., 202) of the actuatorsystem 204. For example, in some implementations, the position sensor208 can determine the position of the motor 212 using a multi-polemagnetic strip. The multi-pole magnetic strip can be located on anoutside of the motor 212, a rotor of the motor 212, a stator of themotor 212, and/or any other location that allows the multi-pole magneticstrip to determine the position of the motor 212. In some cases, themulti-pole magnetic strip can sit flush along the outside of the rotorof the motor 212.

In some examples, when generating control signals for the motor 212, themotor controller 240 can translate the control signals into a format andpower level that can move the motor 212 to a specific position. Thespecific position can be defined in the one or more control signals aspreviously explained. The motor controller 240 can transmit thetranslated signals to the motor 212 in order to move the motor 212 tothe specific position. Based on the translated signal from the motorcontroller 240, the motor 212 can move the sensor carrier structure 220containing the sensors 104-108 in order to move or reposition thesensors 104-108 to the specific position.

In some examples, the motor controller 240 can be electrically coupledwith a fuse box 242. The fuse box 242 can control the electrical flowand power to the motor controller 240. Moreover, the motor controller240 can be communicatively connected to the internal computing system110. The internal computing system 110 and the motor controller 240 canthus communicate data (e.g., instructions, commands, signals, sensordata, motor repositioning data, requests, information, content, etc.) toeach other. In some cases, the motor controller 240 can send and/orreceive data (e.g., instructions, commands, signals, sensor data, motorrepositioning data, requests, information, content, etc.) to and/or fromother devices through the internal computing system 110. For example,the motor controller 240 can send and/or receive data from sensors(e.g., 104-108), a remote computing system (e.g., 150), and/or any otherremote device or location, through the internal computing system 150.Here, the internal computing system 150 can relay such data to and/orfrom the motor controller 240. In other cases, the motor controller 240can communicate directly (or without going through the internalcomputing system 110) with other remote devices or locations.

In some examples, the motor controller 240 can include a communicationinterface that supports network communications to allow the motorcontroller 240 to communicate over one or more networks, such as aprivate network (e.g., a LAN), a public network (e.g., a WAN, a cloudnetwork, etc.), a hybrid network, etc. For example, the motor controller240 can include a communication interface that supports wirelesscommunications, such as cellular communications, WIFI communications,etc.; wired or cable communications, such as Ethernet communications,fiber optic communications, etc.; and/or any other type ofcommunications.

The sensor carrier structure 220 can be attached, coupled, or otherwisesecured to the base 202 in a manner that allows the sensor carrierstructure 220 to rotate and/or move relative to the base 202. Moreover,the sensors 104-108 can be attached, coupled, fixed, or otherwisesecured to the sensor carrier structure 220 via a coupling or securingcomponent, such as a sensor bracket 222. In some examples, the sensors104-108 can be co-located on the sensor carrier structure 220. Thus, bymoving or repositioning the sensor carrier structure 220, the motor 212can also move or reposition the sensors 104-108 on the sensor carrierstructure 220. Also, by affixing and/or co-locating the sensors 104-108on the sensor carrier structure 220, any need to calibrate the sensors104-108 or monitor their relative position can be reduced or eliminated,as the position (actual and relative) of the sensors 104-108 can befixed and known.

The sensor carrier structure 220 can include, for example and withoutlimitation, an articulating or positioning stage, frame, or platform forthe sensors 104-108. For example, the sensor carrier structure 220 canbe an azimuth positioning stage for the sensors 104-108. Moreover, insome examples, the sensor carrier structure 220 can be attached,coupled, fixed or otherwise secured to the actuator system 204.

In some cases, the base 202 and/or the sensor carrier structure 220 canbe attached, coupled, fixed, placed, or otherwise secured to an externalportion of the autonomous vehicle 102 to provide the sensors 104-108access to, and/or visibility into, the outside or external environment.For example, the base 202 and the sensor carrier structure 220 can besecurely placed on a pillar, such as an A-pillar, of the autonomousvehicle 102. In this example, the base 202 and the sensor carrierstructure 202 can reside on an outside of the autonomous vehicle 102between the windshield, the hood of the autonomous vehicle 102, and thepassenger or driver's side. Thus, the sensors 104-108 can reside outsideof the autonomous vehicle 102 and have access to, and/or visibilityinto, the outside or external environment.

In other cases, a portion of the base 202 and/or the sensor carrierstructure 220 can be attached, coupled, fixed, placed, or otherwisesecured to an internal portion of the autonomous vehicle 102, withanother portion of the base 202 and/or the sensor carrier structure 220extending, extruding, protruding, projecting and/or sticking out fromthe autonomous vehicle 102 to an outside of the autonomous vehicle 102.This way, the sensors 104-108 can reside outside of the autonomousvehicle 102 and thus have access to, and/or visibility into, the outsideor external environment.

The motor 212 can move the sensor carrier structure 220 and the sensors104-108 any number of times as previously described, in order to adjustthe position or angle of the sensors 104-108 as desired and thus thevisibility and/or coverage of the sensors 104-108. For example, themotor 212 can move the sensor carrier structure 220 and the sensors104-108 as requested, periodically (e.g., at specific or random timeintervals), randomly, and/or in response to one or more events, such asa maneuver of the autonomous vehicle 102, a change in position or motionof the autonomous vehicle 102, a detected human or object (e.g., anothervehicle, a traffic sign, an object on the road, a guardrail, etc.), adetected condition (e.g., a condition of the autonomous vehicle 102, acondition of the external environment, a traffic condition, a roadcondition, a safety condition or threat, etc.), a navigationinstruction, a predicted navigation event, etc.

The actuator system 204 can include bearings 210 to support movement of,and reduce friction between, one or more moving parts of the motor 212,such as a rotor and a stator. The bearings 210 can also provideincreased axial, radial, and moment load capacity to the motor 212.Moreover, the bearings 210 can be in contact with one or more elementsor portions of the motor 212, as further described herein.

In some examples, the actuator system 204 can also include a shaft seal214 to seal rotary elements (and/or elements in relative motion) in theactuator system 204, such as the motor 212, a shaft of the motor 212, arotor of the motor 212, a rotating bore of the motor 212, etc. In somecases, the shaft seal 214 can be located between the sensor carrierstructure 220 and the actuator system 204. In other cases, the shaftseal 214 can be located between the actuator system 204 and the base 202and/or between the sensor carrier structure 220 and the base 202.

In some implementations, the actuator system 204 can optionally includea brake 206. The brake 206 can be configured to hold and/or control amovement of the motor 212. In some cases, the brake 206 can beconfigured to control and/or manage a holding torque of the motor 212.Moreover, in some examples, the brake 206 in the actuator system 204 canbe implemented below the motor 212 and the position sensor 208.

In some implementations, the base 202 can house the actuator system 204and the rotary cable assembly 216 and can have a small and/orcylindrical form factor. In other examples, the base 202 can have anyother size, shape or design. Moreover, the base 202 can have one or morehollow sections, such as a hollow shaft, for cables to pass through(e.g., from the bottom and through the middle of the assembly) theassembly to the rotary cable assembly 216, to the top of the base 202,and/or to the sensors 104-108 on the sensor carrier structure 220.

In some cases, one or more of the electronic components or hardware inthe base 202 and/or the actuator system 204 can be implemented by one ormore printed circuit boards (PCBs) or electronic circuits. Moreover, insome examples, the base 202 and/or the actuator system 204 can include amemory or storage device for storing data, a power supply for poweringelectronic components, a communication interface for communicating withother devices, and/or one or more processing components.

In some implementations, the sensor positioning platform 200 can includea surround view camera 230. The surround view camera 230 can be includedin, mounted on, coupled with, or otherwise secured to the base 202 ofthe sensor positioning platform 200. In some cases, the sensorpositioning platform 200 can also include cleaning systems 218A-B forcleaning one or more of the sensors 104-108 on the sensor carrierstructure 220. For example, the sensor positioning platform 200 caninclude a liquid cleaning system 218A and an air cleaning system 218Bfor using liquid and air to clean image sensors (e.g., 106, 108) on thesensor carrier structure 220. The liquid cleaning system 218A and theair cleaning system 218B can each include a discharge element such as anozzle, vent, or spraying device for controlling and enabling the flow,discharge, and/or projection of liquid and/or air to the sensors on thesensor carrier structure 220.

The liquid cleaning system 218A and the air cleaning system 218B canalso include a hose, pipe, tube, enclosed chamber, or enclosed carrierelement, which can be attached, coupled, connected, affixed, or securedto the discharge element and can carry, provide, and/or direct liquidand air to the liquid cleaning system 218A and the air cleaning system218B. The discharge elements in the liquid cleaning system 218A and theair cleaning system 218B can receive liquid and air from theirrespective hoses, pipes, tubes, enclosed chambers, or enclosed carrierelements, and can output (e.g., discharge, spray and/or project) thereceived liquid and air towards sensors on the sensor carrier structure220 in order to clean those sensors.

In some implementations, the liquid cleaning system 218A and the aircleaning system 218B can be positioned on a stationary portion of thesensor positioning platform 200, such as a stationary portion of thebase 202, as further described herein. The liquid cleaning system 218Aand air cleaning system 218B can output the liquid and air when thesensors 104-108 on the sensor carrier structure 220 move or rotatewithin an output range of the liquid cleaning system 218A and aircleaning system 218B. In some examples, the liquid cleaning system 218Aand air cleaning system 218B can output the liquid and air as sensorsrotate or move within an output range.

In other examples, the actuator system 204 can move the sensors (e.g.,by moving the sensor carrier structure 220) within an output range ofthe liquid cleaning system 218A and air cleaning system 218B when asensor cleaning operation is to be performed. In some cases, theactuator system 204 can also move or rotate such sensors several timeswithin a range of the liquid cleaning system 218A and air cleaningsystem 218B to change the positioning of the sensors, increase acoverage of the cleaning liquid and air on the sensors and/or ensureoptimal cleaning of the sensors.

In other implementations, the liquid cleaning system 218A and aircleaning system 218B can be positioned on a rotating portion of thesensor positioning platform 200, such as the sensor carrier structure220, as further described herein. The liquid cleaning system 218A andthe air cleaning system 218B can be positioned on the rotating portionat a respective location relative to the sensors 104-108 on the sensorcarrier structure 220. The respective location can be within an outputrange that allows liquid and air outputted by the liquid cleaning system218A and air cleaning system 218B to reach and clean sensors on thesensor carrier structure 220.

In some examples, the motor controller 240, the fuse box 242, thecommunications device 244, and/or the image processing engine 246described above can be part of, or implemented by, the sensorpositioning platform 200. In other examples, the motor controller 240,the fuse box 242, the communications device 244, and/or the imageprocessing engine 246 described above can be separate from the sensorpositioning platform 200.

While the sensor positioning platform 200 and the actuator system 204are shown in FIG. 2 to include certain components, one of ordinary skillwill appreciate that the sensor positioning platform 200 and/or theactuator system 204 can include more or fewer components than thoseshown in FIG. 2 . For example, in some instances, the sensor positioningplatform 200 and/or the actuator system 204 can include one or moredifferent or additional components such as one or more memory components(e.g., one or more RAMs, ROMs, caches, buffers, and/or the like), one ormore processing devices that are not shown in FIG. 2 , one or moretransistors, one or more data communication components (e.g., networkinterfaces, communication devices, antennas, etc.), one or more storagedevices (e.g., one or more hard drives, one or more solid-state drives,and/or the like), one or more circuits that are not shown in FIG. 2 ,one or more sensors that are not shown in FIG. 2 , and/or any otherelectronic or mechanical component.

FIG. 3 is an example method of vehicle localization. A person ofordinary skill in the art would recognize that additional and/oralternative localization methods can be implemented. As shown in FIG. 3, a method 300 for image-based vehicle localization includes measuringvehicle position and heading S310, imaging a road surface S330,processing road image data S340, and generating a local map S350. Themethod 300 may additionally or alternatively include receivingpre-existing map data S320, generating control feedback S360, generatingdriver feedback S370 and/or updating global map data S380.

The method 300 can function to enable localization of a vehicle relativeto its surroundings (e.g., lane markers or other road surface markings)to increase safety, accuracy, and performance for driver assistance andvehicle automation systems.

The method 300 can be operable using an image capture and processingsystem coupled to a motor vehicle. The method 300 can be intended forimplementation with mass-manufactured automobiles, but may additionallyor alternatively be implemented in any suitable motor vehicle; forexample, watercraft, light machinery, motorcycles, etc. In the presentapplication, examples are presented under the assumption ofimplementation in an automobile. A person of ordinary skill in the artwill recognize that the details of these examples may be modified foralternative implementations. The following paragraphs will assumeoperating of the method 300 in an image capture system coupled to anautomobile (i.e., ‘the car’).

The image capture system operating the method 300 can include afront-facing camera and a rear-facing camera, but may additionally oralternatively include only a single one of these cameras, additionalcameras, and/or other image capture means. The image capture system caninclude RGB visible light cameras, but may additionally or alternativelyinclude any suitable image capture devices, including monochrome visiblelight cameras, infrared cameras, depth cameras, etc.

The image capture system can also include a computing device forperforming image processing, but may additionally or alternativelyoffload some or all computational duties to remote servers (e.g., in thecloud) or any other suitable computers.

The image capture system may be coupled to any sensors that may aid inor be relevant to vehicle localization, e.g., photodetectors, RADAR,SONAR, LIDAR, GPS, inertial measurement units (IMUs), microphones,strain gauges, pressure monitors, barometers, thermometers, altimeters,etc.

The image capture system may additionally or alternatively be coupled toany vehicle control systems aiding in vehicle automation, includingsteering, brake, and/or throttle interfaces.

The image capture system may additionally or alternatively be coupled toany user control interfaces (e.g., buttons, touchscreens, knobs, etc.).

S310 can include measuring vehicle position and heading. S310 caninclude measuring vehicle position and heading using a Global NavigationSatellite System (GNSS) receiver (e.g., GPS receiver), but mayadditionally or alternatively include measuring vehicle position and/orheading in any manner.

S310 can include measuring global position (e.g., latitude andlongitude), but may additionally or alternatively include measuringrelative position (e.g., distance and orientation relative to areal-time kinematic (RTK) GNSS base station, displacement from a radiotower, etc.).

S310 can include measuring position using standard GNSS navigationtechniques, but may additionally or alternatively include measuringposition using RTK GNSS techniques, triangulation/trilaterationtechniques (e.g., using cellular towers), visual techniques (e.g.,matching images to images with known locations), and/or any othersuitable techniques. For example, S310 may include measuring positionusing dead-reckoning techniques (e.g., using accelerometer data toestimate change in position from a previous location with knownposition).

S310 can include measuring heading using standard GNSS techniques, butmay additionally or alternatively include measuring heading using a GNSSCompass technique (i.e., using two receivers separated by a knowndistance to generate heading), using a magnetic compass, using agyrocompass, using an accelerometer, inferring heading from change inposition, using visual data, and/or in any other manner.

S310 can include measuring global heading (e.g., degrees from magneticnorth) but may additionally or alternatively include measuring relativeheading (e.g., heading relative to a roadway, heading relative to othercars, etc.).

In one example, S310 can include determining heading relative to ahighway lane by measuring distortion of a road feature (e.g., a lanemarker) as imaged by camera mounted on the vehicle. For example, angleof a lane marker may correspond to an angle of the vehicle relative to aroad (or a lane in that road). S310 may additionally or alternativelyinclude determining heading relative to any road feature, landmark, orother object.

S310 may additionally or alternatively include measuring vehiclevelocity and/or speed. Step S310 can include measuring speed bymeasuring position over time (e.g., as determined using GNSS methods),but may additionally or alternatively include measuring speed in anymanner (e.g., by integrating acceleration as measured by theaccelerometer over time, by querying a vehicle speedometer).

S320 can include receiving pre-existing map data. S320 functions toreceive pre-existing data (i.e., data existing prior to execution ofS320) describing the area the vehicle is traveling in.

Pre-existing map data may include any data relevant to an area around ornear vehicle position. Pre-existing map data can include a map of roads(e.g., the roads the vehicle is traveling on) with correspondinglocation information, but may additionally or alternatively includeweather data, altitude data, point of interest data, etc. Pre-existingmap data may additionally or alternatively include any informationrelevant to mapping; for example, pre-existing map data may includeimages of a road surface linked to locations along the road (e.g., a setof images taken at particular positions and orientations along the road)and/or LIDAR maps of a given area.

Pre-existing map data may be used for a number of purposes. For example,pre-existing map data may be used to aid in the generation of the localmap (S350). In one example, pre-existing map data can be used to providea first estimate of location, which is then refined during generation ofthe local map. In this example, location may be estimated by, forinstance, comparing image data to pre-existing image data (e.g., of abuilding, of the road surface, etc.) known to correspond to a particularlocation. Alternatively, pre-existing map data may be used to refine anestimate of location. For example, the method 300 may include receivingan initial estimate of position (e.g., via GPS), requesting andreceiving pre-existing data corresponding to the position estimate(e.g., image data linked to positions within a region surrounding theposition estimate), and using the pre-existing data (e.g., by comparingit to current road surface images and receiving the linked position datafor matching images), providing a more accurate position estimate thanvia the first estimate alone. S320 may include processing data in anymanner (e.g., there may not be a precise match between a road surfaceimage and a pre-existing road surface image, but image warpingtechniques can be used to correct for differences in perspective; thesetechniques can also identify a correction to position from the positionlinked to the pre-existing road surface image based on the warpingtransformation required to match the perspective of the current roadsurface image).

Pre-existing map data may also be used to calibrate sensors. Forexample, a set of road surface images may be linked to a precisely knownlocation. This map data may be used to calibrate a GPS receiver of thevehicle (i.e., when the vehicle image sensor captures an imagesubstantially similar to one of the set of road surface images, the GPSreceiver can be calibrated using the precisely known location linked tothat road surface image).

Pre-existing map data may be used for comparison purposes; for example,to track changes in an environment over time. Comparison of pre-existingmap data to current map data may provide insight into myriaddriving-related factors; for example, commercial/residential/industrialdevelopment, road surface conditions, weather conditions, trafficconditions, etc. For example, the method 300 may include comparingcurrent data to pre-existing data to identify if a business has movedbased on signs, building lighting, foot traffic, etc.

Pre-existing map data may also be used to supplement the local map. Forexample, pre-existing map data may be used as an alternative to localmap data outside of a certain range (e.g., beyond a mile from thevehicle). As another example, pre-existing map data may be used to addinformation to the local map (e.g., gas station or other point ofinterest locations).

In one example, pre-existing map data can be used in concert with imagedata from a rear-facing camera to provide predictions of local maps(e.g., lane markers) to the front of the vehicle. In this example, therear-facing camera image data may be used (independently or with anotherlocalization technique such as GPS) to localize the vehicle (eitherglobally or locally) and retrieve information about local maps (frompre-existing map data) in front of the vehicle (which can be used tonavigate the vehicle within a lane, for instance). Alternatively, therear-facing camera image data may be used with models that predict thelane boundaries in front of the vehicle and be based on image data takenfrom the rear of the vehicle.

Pre-existing map data can be used when available, but may additionallyor alternatively not be used (e.g., if GPS data is not available) forany reason.

S330 can include imaging a road surface. S330 functions to capture imagedata that may be used for vehicle localization purposes. S330 mayadditionally or alternatively include imaging other aspects of thevehicle's surroundings (e.g., other cars, street signs, buildings on theside of the road), image data of which may be used for vehiclelocalization, guidance, and/or other purposes. Note that image data asreferred to throughout this specification may include any image and/orpseudo-image data as appropriate (e.g., RGB image data, infrared imagedata, LIDAR data, radar density estimate data, depth camera data, etc.).

S330 can include imaging the road surface with at least two cameras; onefront-mounted camera and one rear-mounted camera. Additionally oralternatively, S330 may include imaging the road surface with any numberof cameras mounted in any manner and location. Cameras can be configured(through positioning, focus, lens choice, etc.) to provide road surfaceimage data for road surface between five and fifty feet from thevehicle; additionally or alternatively, cameras may capture road imagedata of road surfaces at any distance or range of distances from thevehicle.

S330 can include capturing images at 30 frames per second or higher;alternatively, S330 may include capturing images at any rate. If S330includes capturing images from multiple cameras, S330 may includesynchronizing cameras to capture images at substantially same times;alternatively, cameras may be unsynchronized.

S330 can include capturing visible light color images; additionally oralternatively, S330 may include capturing any manner of image (e.g.,monochrome visible light, infrared, depth map, etc.).

S330 can include capturing two-dimensional images, but may additionallyor alternatively include capturing three-dimensional images (e.g., viastereo, assisted stereo, structured light, and/or time of flighttechniques).

S340 can include processing road image data. S340 functions to transformroad image data captured in S330 (potentially in combination with otherdata) into localization, guidance, and/or mapping data.

S340 can include one or more of performing image distortion correctionS341, performing image filtering S342, performing image data fusionS343, performing image stitching S344, and performing feature detectionS345. S340 may additionally or alternatively include any otherprocessing of road image data or related data.

S341 can include performing image distortion correction. S341 functionsto transform image data in response to image distortion occurring fromcamera configuration (e.g., distortion from a wide-angle lens), vehiclemotion, vehicle orientation, and/or other imaging system/environmentalproperties.

S341 can include performing image distortion correction in order toincrease accuracy of image-based localization and/or guidance; forexample, if lane markers are imaged improperly, distance from themarkers or orientation relative to the markers may be misjudged.

In one example, S341 can be used to correct pitch and roll error; thatis, image distortion due to vehicle roll (e.g., due to turning) orvehicle pitch (e.g., due to acceleration or braking). S341 mayadditionally or alternatively be used to correct any type of imagedistortion; e.g., image distortions due to lensing, reflection,refraction, sensor noise, and/or occlusions on camera lenses.

S341 may include performing image distortion correction with a number oftechniques, including external-data-based techniques, model-basedtechniques, and/or multi-image-based techniques.

External-data-based techniques can include techniques that make use ofdata external to image data to correct image data. For example, S341 mayinclude performing pitch/roll correction for images based on data froman inertial measurement unit (IMU) or on other data that may becorrelated to pitch and/or roll (e.g., acceleration/deceleration asmeasured by a GPS receiver or by internal vehicle sensors).

Model-based techniques include techniques that make use of image models;that is, sets of data that describe how an image (or features within animage) should look. For example, a model for lane markers may includeparameters for the shape of lane markers as well as for cameraconfiguration and location. If an imaged lane marker deviates from themodel, this may be an indication of image distortion.

Model-based techniques may include comparing image data to static models(e.g., static lane marker proportion data) or to dynamic models (e.g.,models developed or updated by machine learning techniques).

Model-based techniques may take as input any number of parameters, forinstance, an example dynamic model may take as input an estimated pitchand roll (e.g., from an IMU), and refine the pitch and roll estimatesthrough a minimization process (or other optimization process) operatingon an imaged lane marker and the output of the model (e.g., the firstestimate of pitch and roll is the starting point for the model, and themodel iteratively refines the pitch and roll estimates to make the modeloutput most closely match the detected feature).

Multi-image techniques can include techniques that make use of multipleimages to perform distortion correction. While the previous techniquesmay be used on single or multiple images, multi-image techniques arespecifically techniques that rely on the relationship between imagescaptured at successive times in order to perform image correction. Forexample, multi-image techniques may include averaging over several imageframes to remove small pitch/roll errors. As another example,multi-image techniques may include comparing portions of an image to asuccessive image; rapid, large changes in an image may be an indicationof distortion or sensor error.

S341 may include performing image distortion correction from anycombination of techniques; for example, S341 may include correctingimage distortion based on a model that describes lane marker motion (asdetected over multiple image frames) based on external data. In thisexample the model may expect lane markers to be transformed in aparticular way across image frames based on the detected speed of thevehicle, steering wheel position (or other measure of orientation, suchas wheel angle of the wheels used to steer the car) and the frame rateof the camera.

In some examples, a dynamic model used can calculate an expected timedistortion contribution; i.e., how much it is expected that model outputwill change over time based on some parameters, such as vehicle speed,steering wheel angle, vehicle wheel angle, accelerometer data, etc.

S341 can include transforming images to correct image distortion, butmay additionally or alternatively include simply generating datadescribing distortion present in an image and/or possible corrections.For example, a distorted image may be processed by appending or linkinga dataset describing distortions present within the image withoutmodifying the original image data.

S341 may be performed at any time using any suitable image data,depending on the technique used. For example, an averaging technique maybe performed on raw image data immediately after image capture. Asanother example, a technique that performs image distortion correctionbased on lane marker models may be performed after feature detection(which may identify lane markers within images).

In some cases, S341 may be performed concurrently with or as part ofother tasks (e.g., distortion may be corrected for as part of aminimization algorithm for performing vehicle localization in S350).

S342 can include performing image filtering. S342 functions to enableremoval (or ignorance) of undesired features, properties, or occlusionswithin images. For example, S342 may include detecting anon-road-surface (e.g., a car) in a rear-view camera and filtering outthat surface from the image (e.g., by designating that the road-surfacefeature detector ignore that section of the image). S342 may includeperforming any type of image filtering; for example, S342 may includeperforming optical flow filtering to identify objects within camera viewthat are traveling at non-zero speed relative to the road surface (e.g.,a car, a moving object in the road, etc.).

S343 can include performing image data fusion. S343 functions to fuseimage data from multiple sensors to produce new image data; for example,S343 may include using stereo techniques to generate depth informationfrom two front-mounted cameras. As another example, S343 may includecorrelating LIDAR range data to a 2D visible light image to generate 3Dimage data. As a third example, S343 may include fusing an infraredimage and a visible light image to form a hyperspectral image. As afourth example, S343 may include averaging image data between twocameras placed closely together (e.g., to reduce sensor noise). S343 mayinclude performing image fusion in any manner.

S344 can include performing image stitching. S344 functions to combinemultiple image frames to generate a larger image. This larger image maybe useful in generating a local map (as in S350).

S344 can include performing image stitching based on one or acombination of pixel matching, feature matching, and external-data-basedalignment.

Pixel matching refers to an image stitching technique that aligns imagesbased on an algorithm that attempts to minimize the difference in pixelvalues in overlapping image regions. Feature matching refers to an imagestitching technique that includes performing feature detection (as inS345) and aligning images based on detected features.External-data-based alignment refers to using non-image data to performalignment; for example, an overlap between two images taken successivelymay be estimated using change in vehicle position (e.g., as measured byvelocity) between the two images. External-data-based alignment, ifused, can be used as a first estimate for other image stitchingtechniques (but may additionally or alternatively be used in anymanner).

S344 can be performed substantially in real-time, but may additionallyor alternatively be performed at any time (e.g., in some cases it may beused for global map generation but not for local map generation). S344can be performed using temporally successive images, but mayadditionally or alternatively be used with images of a location taken inthe past. For example, if a panorama of road surface is desired but aparticular piece of image data is missing, image data taken at previoustimes may be used to fill in gaps (or for any other purpose).

S344 may be performed per camera (i.e., image stitching occurs forimages taken from one perspective) but may additionally or alternativelybe performed across cameras (i.e., image stitching occurs for imagestaken from multiple cameras and/or perspectives). In some cases, arear-facing camera may be used to provide image data for localizationand/or navigation as a supplement or as a replacement to image data froma front-facing camera (e.g., if the front-facing image is washed out dueto lighting conditions).

In some cases, S344 may be performed concurrently with or as part ofother tasks (e.g., feature detection of S345 may be used to performimage stitching).

S345 include performing feature detection. S345 functions to detectfeatures present in image data captured in S330. S345 can includedetecting lane markers on road surfaces, but may additionally oralternatively include detecting any image feature (e.g., cars, streetsigns, potholes, reflections, etc.).

Features detected by S345 can be used to aid in generating the local mapof Step S350, but may additionally or alternatively be used in any othermanner (e.g., in correcting for pitch/roll error).

In one embodiment, S345 can include performing feature detection forlane markers using multiple binary feature detectors.

For example, S345 may include, for each pixel in an image (or for asubset of pixels; e.g., pixels above some absolute threshold intensityor pixels in an area of interest), measuring difference in intensitybetween the pixel and a pixel some number of pixels to the left. Thisdistance (i.e., the number of pixels) will henceforth be referred to asdx. Dx can correspond to the width of a lane marker as imaged by thecamera (e.g., a number of pixels corresponding to a width of 4-6inches). If the intensity difference passes a threshold value (that cancorrespond to the difference in intensity between road surface and lanemarker paint), the feature is detected; that is, the pixel is identifiedas potentially belonging to a lane marker. This process also can beperformed based on the difference in intensity between the pixel andanother pixel; this other pixel located a distance of dx to the right.Note that the pixel distance dx can be computed using a transformationthat accounts for the parameters of the imaging system (e.g., the 6-axisposition and orientation of the imaging system relative to the vehicleand relative to the road surface); for example, at the bottom of animage (i.e., closest to the vehicle for a downward angled camera), dxfor a lane marker width of four inches may be 200 pixels, while in themiddle of the image (depending on, for example, the angle of the roadsurface relative to the camera), dx for a lane marker width of fourinches may be only 50 pixels.

Further note that intensity differences may be signed (e.g., potentiallycausing the feature detector operating on a white pixel with a blackpixel dx away may result in a true value, but operating on a black pixelwith a white pixel dx away may result in a false value) or unsigned.

The method 300 may additionally or alternatively include modifying thefeature detectors to account for changing road surface slope (i.e.,grade); for example, the method 300 may determine from one set of localimage data (e.g., LIDAR) that road slope changes within the imaging view(e.g., of an RGB camera) and may calculate dx based not just on positionwithin the imaging view, but also on known road slope (or road slopechange).

In this example, S345 may include repeating this process for distancesof 2dx to the left and the right (or any other distance value). In thiscase, the different distances may be assigned different confidencelevels; that is, the 2dx process may be more likely to detect lanemarker pixels, but may also be more likely to generate false positiveresults (than the dx process). Further, S345 may include using avertical feature detector (in contrast to the horizontal detectorsdescribed in the preceding two paragraphs) to provide additionalconfidence to feature detection; for instance, a set of pixels thatsatisfy the horizontal feature detection constraints may be eliminatedif they are not vertically near other pixels satisfying thoseconstraints (the vertical axis of the image corresponding to thedirection of travel of the car). From these feature detectors, a set ofbinary maps may be generated for both dx and 2dx, which may then be usedin local map generation.

Note that for multiple feature detectors (operating with multiple‘width’ parameters as described above), it may be possible to generatescaled pixel ‘widths’ simply by scaling the first generated pixel width(as opposed to scaling the actual width and calculating the transformedpixel width directly for each width). Alternatively, pixel widths may begenerated in any manner.

S345 may additionally or alternatively include using region-growingtechniques to identify features. For example, the two binary mapsdiscussed above may be merged using a region-growing technique, whereinareas of low-confidence pixels are only considered as feature areas ifthey couple or overlap with areas of high confidence pixels.

S345 may include detecting image features using any other suitablefeature detection techniques (e.g., color matching, edge/corner/blobdetectors, machine-learning-based techniques). Image feature detectiontechniques need not necessarily be binary (e.g., in the dx example, apixel could receive a scaled score based on intensity difference ratherthan a binary value).

In one example, S345 can include generating expected features (e.g.,expected lane markers at particular locations) based on previouslycaptured data (e.g., recently captured road surface data or the‘pre-existing data’ received in S320). These predicted features may inturn be used to more efficiently search for features in S340.

The parameters of feature detection may vary based on externalparameters. For example, S345 may include lowering feature detectionthresholds if it is cloudy outside (e.g., as determined by weather, aphotodetector, etc.). This may be determined, for example, by measuringthe average intensity of an image. As another example, S345 may includealtering feature detection thresholds based on vehicle speed.

S350 can include generating a local map. S350 functions to create a mapof features in proximity to a vehicle. S350 can include generating alocal map of lanes in front of a vehicle to aid in assisted cruisecontrol, but may additionally or alternatively generate local map dataof any features for any purpose.

For example, S350 may include imaging road surface near the vehicle andcorrelating this to a global map (e.g., via latitude and longitude) sothat road surface images may be used to supplement GPS data on futuretrips (or for other cars, etc.). Note that the term “global map” hererefers to a map of features (or any other map) extending beyond acertain proximity of the vehicle; for example, a city street map. Globalmaps can be referenced to a fixed point, while local maps can bereferenced to the vehicle. Additionally or alternatively, global andlocal maps may be referenced to any coordinates (or may be referenced inany other manner).

S350 can include generating a local map of road features (e.g., lanes)from features detected in images captured by S330, but may additionallyor alternatively include generating a local map using any suitable data.For example, S350 may include generating a local map based on acombination of image data and GPS data (and/or IMU data).

In one example of an invention embodiment, S350 can include identifyinga lane demarcated by detected lane markers (e.g., by interpolatingbetween lane markers).

S350 can include generating local map data by converting pixel offsetsinto distances (using known camera configuration parameters, 3D imagingtechniques, or any other suitable method).

S350 can include generating local map data in a manner robust topitch/roll error. For example, S350 may include generating a local mapstarting from the bottom of the image frame (that is, the area of theimage frame closest to the vehicle), which is generally less susceptibleto pitch error. S350 may additionally or alternatively includeintegrating other techniques for reducing pitch/roll/yaw error (such asthose described previously).

In one example, S350 can include generating a local map identifying alane by computing estimated lane boundaries for several successive (orotherwise captured-close-in-time) image data frames and determining thelane boundaries (relative to the vehicle) from an average of themultiple lane boundary estimates.

In one example, S350 can include generating a local map using a Kalmanfilter operating on processed image data and GPS receiver data. S350 mayadditionally or alternatively include generating a local map using anysuitable technique, such as utilizing one or more of: supervisedlearning (e.g., using logistic regression, using back propagation neuralnetworks, using random forests, decision trees, etc.), unsupervisedlearning (e.g., using an Apriori algorithm, using K-means clustering),semi-supervised learning, reinforcement learning (e.g., using aQ-learning algorithm, using temporal difference learning), and any othersuitable learning style. Each module of the plurality can implement anyone or more of: a regression algorithm (e.g., ordinary least squares,logistic regression, stepwise regression, multivariate adaptiveregression splines, locally estimated scatterplot smoothing, etc.), aninstance-based method (e.g., k-nearest neighbor, learning vectorquantization, self-organizing map, etc.), a regularization method (e.g.,ridge regression, least absolute shrinkage and selection operator,elastic net, etc.), a decision tree learning method (e.g.,classification and regression tree, iterative dichotomiser 3, C4.5,chi-squared automatic interaction detection, decision stump, randomforest, multivariate adaptive regression splines, gradient boostingmachines, etc.), a Bayesian method (e.g., naïve Bayes, averagedone-dependence estimators, Bayesian belief network, etc.), a kernelmethod (e.g., a support vector machine, a radial basis function, alinear discriminate analysis, etc.), a clustering method (e.g., k-meansclustering, expectation maximization, etc.), an associated rule learningalgorithm (e.g., an Apriori algorithm, an Eclat algorithm, etc.), anartificial neural network model (e.g., a Perceptron method, aback-propagation method, a Hopfield network method, a self-organizingmap method, a learning vector quantization method, etc.), a deeplearning algorithm (e.g., a restricted Boltzmann machine, a deep beliefnetwork method, a convolution network method, a stacked auto-encodermethod, etc.), a dimensionality reduction method (e.g., principalcomponent analysis, partial lest squares regression, Sammon mapping,multidimensional scaling, projection pursuit, etc.), an ensemble method(e.g., boosting, boostrapped aggregation, AdaBoost, stackedgeneralization, gradient boosting machine method, random forest method,etc.), and any suitable form of machine learning algorithm. Eachprocessing portion of the method 300 can additionally or alternativelyleverage: a probabilistic module, heuristic module, deterministicmodule, or any other suitable module leveraging any other suitablecomputation method, machine learning method or combination thereof.However, any suitable machine learning approach can otherwise beincorporated in the method 300. Further, any suitable model (e.g.,machine learning, non-machine learning, etc.) can be used in generatinglocal maps and/or other data relevant to the method 300.

S350 may additionally or alternatively include generating a local mapbased on local map models. For example, local map generation may beconstrained by known road standards (e.g., maximum allowed curvature ofa highway, known turning radii, lane marker separation requirements).

S350 may additionally or alternatively include linking local mapfeatures. For example, features (e.g., lane markers) may be linked toeach other, allowing their presence, location, and/or shape to beinferred. This may be used, for instance, to allow lane markers to beinferred even if they are occluded (e.g., by a car) or otherwisedifficult to detect (e.g., due to lighting issues). Lane markers may beinferred or linked in any manner (e.g., using a model of lane markersthat includes separation between lane markers, allowing probably lanemarker locations to be generated based on detected lane markers).

In some cases, S350 can include generating different local maps fordifferent imaging areas (e.g., corresponding to differing distances).For example, S350 may include mapping multiple road features fordistances less than 30 feet, while only mapping lane markers (orinferring lane markers) for distances greater than 30 feet from thevehicle.

Traffic Object Mapping:

In an autonomous vehicle environment (100), the environment itselfcontinuously changes over time, which may influence updates to local orglobal maps of the autonomous vehicle environment (100). For example,traffic object position, type, and lane association can change abruptlydue to construction or a car accident. Traffic objects can includetraffic signs, traffic signals, traffic lights, traffic cones, vehicles,fire hydrants, bike racks, constructions signs, constructions barriers,construction objects, or any other traffic object suitable for theintended purpose and understood by a person of ordinary skill in theart.

If traffic object maps are not updated accordingly, the autonomousvehicle (102) may not be able to detect a traffic object. For example,according to the autonomous vehicle's (102) current map, a stop signshould be at an intersection, when in actuality, the stop sign has beenreplaced with a traffic light because of construction. Without themethod described below, the autonomous vehicle (102) might enter intothe intersection upon determining it had sufficiently waited at the stopsign, but instead, is running a red traffic light. Alternatively, theautonomous vehicle (102) might perform an emergency stop upon failing todetect the stop sign. As such, a method for detecting changes in trafficobjects is needed to enable autonomous vehicles (102) to performappropriately within a changing autonomous vehicle environment (100).

Traffic object position information also can be seeded with onboardperception models configured to receive live traffic object informationfrom camera data (104-108). Traffic objects can be mapped as priors forautonomous vehicles (102). For example, for a traffic light, if thetraffic light is missing from an expected position, object detectionmodels can be utilized to determine candidates on how to proceed. Thesystem described herein can use the bearing from the candidates and theprevious position of the traffic objects to compute a new position. Inone embodiment, as the traffic object is passed by the autonomousvehicle (102), 2D detections from a “broken” traffic object can beaggregated and data tagged accordingly. Thereafter, the new position ofthe broken traffic object can be determined, processed, and trafficobject maps updated accordingly. Workflows also can automatically updatetraffic object location based on autonomous vehicle (102) detection andtracking. Artificial intelligence and machine learning also can beutilized in the detection of traffic objects and updating of trafficobject maps.

Having disclosed some example system components and concepts, thedisclosure now turns to FIGS. 4A and 4B, which illustrate examplemethods 400A and 400B for managing traffic object discrepancies. Thesteps outlined herein are exemplary and can be implemented in anycombination thereof, including combinations that exclude, add, or modifycertain steps. Embodiments may refer to a traffic light, a trafficsignal, or a traffic sign, but any traffic object as described in thisdisclosure can be utilized in any of the embodiments described herein.

At step 401, the method 400A can include searching for a predeterminedtraffic sign within a predetermined geographic location by analyzing aregion of an image frame associated with the traffic sign. The imageframe can be captured by a localized vehicle. For example, the vehiclecan be localized (i.e., have a known location/orientation) bydetermining a location of an identified traffic sign in a full frameimage. The image frame also can be a partial frame, a full frame image,or any other frame image suitable for the intended purpose andunderstood by a person of ordinary skill in the art.

At step 403, the method 400A can include searching for and identifying anew traffic sign or the predetermined traffic sign within a full frameimage. The method 400A further can include searching and identifying thetraffic sign within a plurality of full frame images.

At step 405, the method 400A can include associating the identifiedtraffic sign with the predetermined traffic sign.

At step 407, the method 400A can include determining a geographiclocation of the identified traffic sign based on localizationinformation of the vehicle that was captured in the image frame.

At step 409, the method 400A can include approximating a variationbetween the geographic location of the identified traffic sign and thepredetermined geographic location of the predetermined traffic sign. Forexample, the step 409 can be performed by determining locations of theidentified traffic sign over a plurality of full frame images anddetermining an average location. The determined average location canthen be used to determine a variation between the identified trafficsign and the predetermined traffic sign.

At step 411, the method 400A can include transmitting a messageindicative of the variation exceeding a threshold. The threshold can bea predetermined threshold or based on localization information of thevehicle. The message can trigger a manual or automated review procedureor cause an automated local or global map update. The vehicle canadditionally/alternatively update an on-board map accordingly.

At step 402, the method 400B can include receiving traffic light datafrom a plurality of autonomous vehicles (102) by a monocular camera(e.g., a single camera that is capable for receiving positional data andangular data), the traffic light data including a geographic location ofa traffic light. The traffic light data can be received by the sensors(104-108) of the autonomous vehicles (102). For example, while the fleetof autonomous vehicles (102) are actively driving, the sensors (104-108)can detect and/or determine the presence of a traffic light and itscorresponding features such as traffic light lane association, angle ofthe traffic light, position of the traffic light, type of the trafficlight, duration of the traffic light, color of the traffic light, or anyother traffic light data suitable for the intended purpose andunderstood by a person of ordinary skill in the art.

The type of the traffic light can include arrow-type traffic lights andarrowless-type traffic lights. For example, at a given time, a trafficlight can include an arrow-style signal. However, at a different time,the traffic light may be replaced or moved such that the traffic lightdoes not include an arrow-style signal. If the fleet of autonomousvehicles (102) is not made aware of this change, the autonomous vehiclesmay proceed into an intersection inadvertently.

At step 404, the method 400B can include comparing the traffic lightdata of each of the plurality of autonomous vehicles (102) with knowntraffic light data. The known traffic light data can be obtained fromthe sensors (104-108) of the fleet of autonomous vehicles (102),internal maps, third party vendor maps, or any other map suitable forthe intended purpose and understood by a person of ordinary skill in theart.

At step 406, the method 400B can include determining a discrepancybetween the traffic light data of each of the plurality of autonomousvehicles (102) and the known traffic light data. The discrepancy betweenthe traffic light data of each of the plurality of autonomous vehicles(102) and the known traffic light data can be a value that correspondsto the difference between the received traffic light data of theplurality of autonomous vehicles (102) and the known traffic light data.For example, the value of the discrepancy can be a binary value (e.g.,0—indicating no change; and 1—indicating a change), a percentage value(e.g., a percentage of change of an angle of the traffic light), a rangevalue (e.g., the degree or extent of change from the previously knowntraffic light data), or any other value suitable for the intendedpurpose and understood by a person of ordinary skill in the art.

At step 408, the method 400B can include grouping the traffic light dataof each of the plurality of autonomous vehicles (102) based on thedetermining of the discrepancy between the traffic light data of each ofthe plurality of autonomous vehicles (102) and the known traffic lightdata. In some instances, one group of autonomous vehicles (102) mayindicate a change in the traffic light, while others do not. Forexample, the autonomous vehicles (102) that have indicated a change inthe traffic light data can be Group A, while the autonomous vehicles(102) that have indicated no change in the traffic light data can beGroup B. Group A and Group B can then be compared to one another and tothe previously known traffic light data to determine which group is moreaccurate. The grouping of the traffic light data also can be based on apercentage of difference between the traffic light data from theplurality of autonomous vehicles (102) and the known traffic light data.For example, 25% of the autonomous vehicles (102) that indicated achange in the traffic light can be in Group A, while 75% of theautonomous vehicles (102) that indicated no change in the traffic lightcan be in Group B. The percentage of difference also can include thedegree of change in the traffic light (e.g., angle of the trafficlight).

At step 410, the method 400B can include determining whether a group oftraffic light data of the grouping of the traffic light data of each ofthe plurality of autonomous vehicles (102) exceeds a threshold. Forexample, the threshold can be a value (e.g., 500 autonomous vehicles(102) indicated a change in the traffic light), percentage (e.g., 30% ofthe autonomous vehicles (102) indicated a change in the traffic light),or any other threshold suitable for the intended purpose and understoodby a person of ordinary skill in the art. Moreover, the determining ofwhether a group of traffic light data exceeds a threshold can bedetermined with artificial intelligence or machine learning to adaptchanges in real-time.

At step 412, the method 400B can include providing a traffic light mapupdate to the plurality of autonomous vehicles (102) based on thetraffic light data of the group that exceeds the threshold. Theproviding of the traffic light map update can be provided to theautonomous vehicles (102) remotely immediately or during scheduledupdates of the autonomous vehicles (102).

In some cases, the method 400B can further include preparing trafficlight data tables that include the traffic light data for each of theplurality of autonomous vehicles (102). The method 400B also can includecomparing the traffic light data tables of each of the plurality ofautonomous vehicles (102) with a known traffic light data table. Themethod 400B can further include grouping the traffic light data tablesbased on a discrepancy level, the discrepancy level being the differencebetween the traffic light data tables of each of the plurality ofautonomous vehicles (102) and the known traffic light data table.

The method 400B also can include replacing the known traffic light datawith the traffic light data of the group that exceeds the threshold,thereby updating the traffic light maps of the fleet of autonomousvehicles (102).

In some examples, the remote computing system (150) can be configured tosend and receive communications and traffic light data from theautonomous vehicles (102) to facilitate the above-mentioned steps ofmethod 400B. For example, the remote computing system (150) can beconfigured to perform the receiving of the traffic light data from theplurality of autonomous vehicles 402, the comparing of the traffic lightdata of each of the plurality of autonomous vehicles with the knowntraffic light data 404, the determining of the discrepancy between thetraffic light data of each of the plurality of autonomous vehicles andthe known traffic light data 406, the grouping of the traffic light dataof each of the plurality of autonomous vehicles based on the determiningof the discrepancy between the traffic light data of each of theplurality of autonomous vehicles and the known traffic light data 408,the determining of whether the group of traffic light data of thegrouping of the traffic light data of each of the plurality ofautonomous vehicles exceeds the threshold 410, and the providing of thetraffic light map update to the plurality of autonomous vehicles basedon the traffic light data of the group that exceeds the threshold 412.

As described herein, one aspect of the present technology includesgathering and using data available from various sources to improvequality and experience. The present disclosure contemplates that in someinstances, this gathered data may include personal information. Thepresent disclosure contemplates that the entities involved with suchpersonal information respect and value privacy policies and practices.

FIG. 5 illustrates an example computing system 500 which can be, forexample, any computing device making up internal computing system 110,remote computing system 150, a passenger device executing rideshareapplication 170, or any other computing device. In FIG. 5 , thecomponents of the computing system 500 are in communication with eachother using connection 505. Connection 505 can be a physical connectionvia a bus, or a direct connection into processor 510, such as in achipset architecture. Connection 505 can also be a virtual connection,networked connection, or logical connection.

In some embodiments, computing system 500 is a distributed system inwhich the functions described in this disclosure can be distributedwithin a datacenter, multiple data centers, a peer network, etc. In someembodiments, one or more of the described system components representsmany such components each performing some or all of the function forwhich the component is described. In some embodiments, the componentscan be physical or virtual devices.

Example system 500 includes at least one processing unit (CPU orprocessor) 510 and connection 505 that couples various system componentsincluding system memory 515, such as read-only memory (ROM) 520 andrandom access memory (RAM) 525 to processor 510. Computing system 500can include a cache of high-speed memory 512 connected directly with, inclose proximity to, or integrated as part of processor 510.

Processor 510 can include any general purpose processor and a hardwareservice or software service, such as services 532, 534, and 536 storedin storage device 530, configured to control processor 510 as well as aspecial-purpose processor where software instructions are incorporatedinto the actual processor design. Processor 510 may essentially be acompletely self-contained computing system, containing multiple cores orprocessors, a bus, memory controller, cache, etc. A multi-core processormay be symmetric or asymmetric.

To enable user interaction, computing system 500 includes an inputdevice 545, which can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech, etc. Computingsystem 500 can also include output device 535, which can be one or moreof a number of output mechanisms known to those of skill in the art. Insome instances, multimodal systems can enable a user to provide multipletypes of input/output to communicate with computing system 500.Computing system 500 can include communications interface 540, which cangenerally govern and manage the user input and system output. There isno restriction on operating on any particular hardware arrangement, andtherefore the basic features here may easily be substituted for improvedhardware or firmware arrangements as they are developed.

Storage device 530 can be a non-volatile memory device and can be a harddisk or other types of computer readable media which can store data thatare accessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs), read-only memory (ROM), and/or somecombination of these devices.

The storage device 530 can include software services, servers, services,etc., that when the code that defines such software is executed by theprocessor 510, it causes the system to perform a function. In someembodiments, a hardware service that performs a particular function caninclude the software component stored in a computer-readable medium inconnection with the necessary hardware components, such as processor510, connection 505, output device 535, etc., to carry out the function.

For clarity of explanation, in some instances, the present technologymay be presented as including individual functional blocks includingfunctional blocks comprising devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

Any of the steps, operations, functions, or processes described hereinmay be performed or implemented by a combination of hardware andsoftware services or services, alone or in combination with otherdevices. In some embodiments, a service can be software that resides inmemory of a client device and/or one or more servers of a contentmanagement system and perform one or more functions when a processorexecutes the software associated with the service. In some embodiments,a service is a program or a collection of programs that carry out aspecific function. In some embodiments, a service can be considered aserver. The memory can be a non-transitory computer-readable medium.

In some embodiments, the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer-readable media. Such instructions can comprise,for example, instructions and data which cause or otherwise configure ageneral purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The executable computer instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware, orsource code. Examples of computer-readable media that may be used tostore instructions, information used, and/or information created duringmethods according to described examples include magnetic or opticaldisks, solid-state memory devices, flash memory, USB devices providedwith non-volatile memory, networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprisehardware, firmware and/or software, and can take any of a variety ofform factors. Typical examples of such form factors include servers,laptops, smartphones, small form factor personal computers, personaldigital assistants, and so on. The functionality described herein alsocan be embodied in peripherals or add-in cards. Such functionality canalso be implemented on a circuit board among different chips ordifferent processes executing in a single device, by way of furtherexample.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

Although a variety of examples and other information was used to explainaspects within the scope of the appended claims, no limitation of theclaims should be implied based on particular features or arrangements insuch examples, as one of ordinary skill would be able to use theseexamples to derive a wide variety of implementations. Further andalthough some subject matter may have been described in languagespecific to examples of structural features and/or method steps, it isto be understood that the subject matter defined in the appended claimsis not necessarily limited to these described features or acts. Forexample, such functionality can be distributed differently or performedin components other than those identified herein. Rather, the describedfeatures and steps are disclosed as examples of components of systemsand methods within the scope of the appended claims.

Claim language reciting “at least one of” a set indicates that onemember of the set or multiple members of the set satisfy the claim. Forexample, claim language reciting “at least one of A and B” means A, B,or A and B.

What is claimed is:
 1. A computer-implemented method comprising:measuring a global position and a global heading of an autonomousvehicle; imaging, by sensors of the autonomous vehicle, a surrounding ofthe autonomous vehicle, the surrounding including at least a roadsurface to obtain road image data; processing the road image data intoat least one of localization data, guidance data, and mapping data; andgenerating a local map of road features surrounding the autonomousvehicle based on the at least one of localization data, guidance data,and mapping data, wherein the local map is correlated to a global map,and wherein the global map is a map of features that extends beyond acertain proximity of the autonomous vehicle.
 2. The computer-implementedmethod of claim 1, further comprising: measuring a relative heading ofthe autonomous vehicle relative to a roadway by measuring distortion ofa road feature as imaged by a sensor of the autonomous vehicle.
 3. Thecomputer-implemented method of claim 1, further comprising: receivingpre-existing map data, wherein the pre-existing map data is used to aidin generating the local map; and refining an initial estimate oflocation by comparing the road image data to linked position data formatching images of the pre-existing map data to provide a more accurateposition estimate than via the initial estimate of location alone. 4.The computer-implemented method of claim 1, wherein refining the initialestimate of location includes image warping techniques to correct fordifferences in perspective.
 5. The computer-implemented method of claim1, further comprising: receiving pre-existing map data, wherein thepre-existing map data is used to aid in generating the local map, andwherein the pre-existing map data provides a first estimate of locationthat is refined during generation of the local map.
 6. Thecomputer-implemented method of claim 1, further comprising: receivingpre-existing map data, wherein the pre-existing map data is used tocalibrate the sensors of the AV based on a set of road surface imageslinked to a precise location.
 7. The computer-implemented method ofclaim 1, further comprising: receiving pre-existing map data; andproviding predictions by using the pre-existing map data in concert witha rear-facing camera.
 8. The computer-implemented method of claim 7,wherein the predictions are lane boundaries in front of the autonomousvehicle based on image data from the rear-facing camera taken from arear of the autonomous vehicle.
 9. A system comprising: a processor; anda non-transitory memory storing computer-readable instructions, whichwhen executed by the processor, cause the processor to performoperations comprising: measuring a global position and a global headingof an autonomous vehicle; imaging, by sensors of the autonomous vehicle,a surrounding of the autonomous vehicle, the surrounding including atleast a road surface to obtain road image data; processing the roadimage data into at least one of localization data, guidance data, andmapping data; and generating a local map of road features surroundingthe autonomous vehicle based on the at least one of localization data,guidance data, and mapping data, wherein the local map is correlated toa global map, and wherein the global map is a map of features thatextends beyond a certain proximity of the autonomous vehicle.
 10. Thesystem of claim 9, wherein the computer-readable instructions, whenexecuted by the processor, further cause the processor to performoperations comprising: measuring a relative heading of the autonomousvehicle relative to a roadway by measuring distortion of a road featureas imaged by a sensor of the autonomous vehicle.
 11. The system of claim9, wherein the computer-readable instructions, when executed by theprocessor, further cause the processor to perform operations comprising:receiving pre-existing map data, wherein the pre-existing map data isused to aid in generating the local map; and refining an initialestimate of location by comparing the road image data to linked positiondata for matching images of the pre-existing map data to provide a moreaccurate position estimate than via the initial estimate of locationalone.
 12. The system of claim 9, wherein refining the initial estimateof location includes image warping techniques to correct for differencesin perspective.
 13. The system of claim 9, wherein the computer-readableinstructions, when executed by the processor, further cause theprocessor to perform operations comprising: receiving pre-existing mapdata, wherein the pre-existing map data is used to aid in generating thelocal map, and wherein the pre-existing map data provides a firstestimate of location that is refined during generation of the local map.14. The system of claim 9, wherein the computer-readable instructions,when executed by the processor, further cause the processor to performoperations comprising: receiving pre-existing map data, wherein thepre-existing map data is used to calibrate the sensors of the AV basedon a set of road surface images linked to a precise location.
 15. Thesystem of claim 9, wherein the computer-readable instructions, whenexecuted by the processor, further cause the processor to performoperations comprising: receiving pre-existing map data; and providingpredictions by using the pre-existing map data in concert with arear-facing camera.
 16. The system of claim 15, wherein the predictionsare lane boundaries in front of the autonomous vehicle based on imagedata from the rear-facing camera taken from a rear of the autonomousvehicle.
 17. A non-transitory computer-readable medium comprisinginstructions stored therein, which when executed by one or moreprocessors, cause the one or more processors to perform operationscomprising: measuring a global position and a global heading of anautonomous vehicle; imaging, by sensors of the autonomous vehicle, asurrounding of the autonomous vehicle, the surrounding including atleast a road surface to obtain road image data; processing the roadimage data into at least one of localization data, guidance data, andmapping data; and generating a local map of road features surroundingthe autonomous vehicle based on the at least one of localization data,guidance data, and mapping data, wherein the local map is correlated toa global map, and wherein the global map is a map of features thatextends beyond a certain proximity of the autonomous vehicle.
 18. Thenon-transitory computer-readable medium of claim 17, wherein theinstructions, when executed by the one or more processors, further causethe one or more processor to perform operations comprising: measuring arelative heading of the autonomous vehicle relative to a roadway bymeasuring distortion of a road feature as imaged by a sensor of theautonomous vehicle.
 19. The non-transitory computer-readable medium ofclaim 17, wherein the instructions, when executed by the one or moreprocessors, further cause the one or more processor to performoperations comprising: receiving pre-existing map data, wherein thepre-existing map data is used to aid in generating the local map; andrefining an initial estimate of location by comparing the road imagedata to linked position data for matching images of the pre-existing mapdata to provide a more accurate position estimate than via the initialestimate of location alone.
 20. The non-transitory computer-readablemedium of claim 17, wherein refining the initial estimate of locationincludes image warping techniques to correct for differences inperspective.